Surgical robot system and method of controlling the same

ABSTRACT

A surgical robot system includes a slave system to perform a surgical operation on a patient and an imaging system that includes an image capture unit including a plurality of cameras to acquire a plurality of affected area images, an image generator detecting an occluded region in each of the affected area images acquired by the plurality of cameras, removing the occluded region therefrom, warping each of the affected area images from which the occluded region is removed, and matching the affected area images to generate a final image, and a controller driving each of the plurality of cameras of the image capture unit to acquire the plurality of affected area images and inputting the acquired plurality of affected area images to the image generator to generate a final image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority benefit of Korean PatentApplication No. 10-2013-0030239, filed on Mar. 21, 2013 in the KoreanIntellectual Property Office, the disclosure of which is incorporatedherein by reference.

BACKGROUND

1. Field

The following description relates to a surgical robot system to restoreoccluded regions, which are hidden regions of a surgical region of apatient, and a method of controlling the same.

2. Description of the Related Art

Minimally invasive surgery generally refers to surgery capable ofminimizing an incision size and recovery time. Whereas laparotomy usesrelatively large surgical incisions through a part of a human body(e.g., the abdomen), in minimally invasive surgery, after forming atleast one small incision hole (or invasive hole) of 0.5 cm to 1.5 cmthrough the abdominal wall, an operator inserts an endoscope andsurgical instruments through the incision hole, to perform surgery whileviewing images provided by the endoscope.

Compared to laparotomy, such minimally invasive surgery causes lesspost-operative pain, faster recovery of bowel movement, earlierrestoration of ability to eat, shorter hospitalization, faster return todaily life, and better cosmetic effects due to the small incision size.Due to these properties, minimally invasive surgery is used forcholecystectomy, prostatic carcinoma surgery, hernia repair, and thelike, and applications thereof continue to grow.

In general, a surgical robot used in minimally invasive surgery includesa master device and a slave device. The master device generates acontrol signal in accordance with manipulation by a doctor and transmitsthe control signal to the slave device. The slave device receives thecontrol signal from the master device and performs manipulation forsurgery of a patient. The master device and the slave device may beintegrated with each other, or may be separately arranged in anoperating room.

The slave device includes at least one robot arm. Surgical instrumentsare mounted on an end of the robot arm.

In such minimally invasive surgery using a surgical robot, surgery isperformed by use of surgical instruments of a slave device which areinserted into a human body. In this regard, the same surgicalenvironment as conventional surgery needs to be provided to amanipulator manipulating a master device. Thus, surgical instrumentscorresponding to arms of an operator are disposed under the operator'sfield of vision. Accordingly, surgical instruments may obstruct the viewof a surgical region.

SUMMARY

Therefore, the following disclosure describes a surgical robot systemcapable of displaying an affected area that does not have occludedregions during surgery and a method of controlling the same.

Additional aspects of the invention will be set forth in part in thedescription which follows and, in part, will be obvious from thedescription, or may be learned by practice of the invention.

In accordance with an aspect of the present disclosure, a surgical robotsystem includes a slave system to perform a surgical operation on apatient and an imaging system that includes an image capture unitincluding a plurality of cameras to acquire a plurality of affected areaimages, an image generator detecting an occluded region in each of theaffected area images acquired by the plurality of cameras, removing theoccluded region therefrom, warping each of the affected area images fromwhich the occluded region is removed, and matching the affected areaimages to generate a final image, and a controller driving each of theplurality of cameras of the image capture unit to acquire the pluralityof affected area images and inputting the acquired plurality of affectedarea images to the image generator to generate a final image.

In accordance with an aspect of the present disclosure, a method ofcontrolling a surgical robot includes acquiring a plurality of affectedarea images using a plurality of cameras, detecting an occluded regionspresent in each of the acquired affected area images, removing thedetected occluded regions from each of the affected area images, warpingeach of the affected area images from which the occluded regions areremoved, and generating a final image by matching the warped affectedarea images.

In accordance with an aspect of the present disclosure, a surgical robotincludes an image capture unit acquiring a plurality of images of anarea from different locations on the surgical robot, and an imagegenerator receiving the acquired images, detecting an occluded region inat least one of the acquired images, removing the occluded region,warping the image from which the occluded region is removed, andmatching the acquired images with the warped image to generate a finalimage.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects of the invention will become apparent andmore readily appreciated from the following description of theembodiments, taken in conjunction with the accompanying drawings ofwhich:

FIG. 1 is a view schematically illustrating a surgical robot system;

FIG. 2 is a block diagram illustrating constituent elements of asurgical robot system;

FIG. 3 is a block diagram illustrating constituent elements of an imagecapture unit of an imaging system;

FIG. 4 is a block diagram illustrating constituent elements of an imagegenerator of an imaging system;

FIG. 5 is a view schematically illustrating a plurality of camerasmounted on a support member;

FIG. 6 is a view illustrating occluded regions generated while asurgical instrument is located between a camera and an affected area;

FIG. 7 is a view illustrating an occluded region generated while asurgical instrument is located close to a camera;

FIG. 8 is a view illustrating occluded regions generated while asurgical instrument is located close to an affected area;

FIG. 9 is a view illustrating occluded regions, location of whichchanges as a camera moves;

FIG. 10 is a flowchart sequentially illustrating a method of controllinga surgical robot system; and

FIG. 11 is a flowchart sequentially illustrating an example of operationS1020 of FIG. 10.

DETAILED DESCRIPTION

The aspects, particular advantages, and novel features of theembodiments of the present invention will become apparent with referenceto the following detailed description and embodiments described below indetail in conjunction with the accompanying drawings. In the drawings,the same or similar elements are denoted by the same reference numeralseven though they are depicted in different drawings. In the followingdescription of the embodiments, a detailed description of knownfunctions and configurations incorporated herein will be omitted when itmay make the subject matter of the embodiments rather unclear. Herein,the terms first, second, etc. are used simply to discriminate any oneelement from other elements, and the elements should not be limited bythese terms.

Hereinafter, the embodiments will be described in detail with referenceto the accompanying drawings.

According to the present embodiment, a surgical robot system including aslave system and a master system that remotely controls the slave systemis exemplarily described. However, any surgical robot system in which anoperator directly controls a slave system may be used.

FIG. 1 is a view schematically illustrating a surgical robot system.FIG. 2 is a block diagram illustrating constituent elements of asurgical robot system. FIG. 3 is a block diagram illustratingconstituent elements of an image capture unit of an imaging system. FIG.4 is a block diagram illustrating constituent elements of an imagegenerator of an imaging system.

Referring to FIG. 1, the surgical robot system may include a slavesystem 200 to perform surgery on a patient P who lies on an operatingtable, and a master system 100 to remotely control the slave system 200in accordance with manipulation by an operator S (e.g., a doctor). Inthis regard, at least one assistant A assisting the operator S may bepositioned near the patient P.

In this regard, assisting the operator S may refer to assisting asurgical task while surgery is in progress, such as replacing surgicalinstruments, for example, but is not limited thereto. For example, avariety of surgical instruments may be used, according to the surgicaltask. Because the number of robot arms 210 of the slave system 200 islimited, the number of surgical instruments 230 mounted thereon at onceis also limited. Accordingly, when the surgical instrument 230 isrequired to be replaced during surgery, the operator S instructs theassistant A positioned near the patient P to replace the surgicalinstrument 230. In accordance with instructions, the assistant A removesa surgical instrument 230 not in use from the robot arm 210 of the slavesystem 200 and mounts another surgical instrument 230′ placed on a trayT on the corresponding robot arm 210.

The master system 100 and the slave system 200 may be separatelyarranged as physically independent devices, without being limitedthereto. For example, the master system 100 and the slave system 200 mayalso be integrated as a single device.

As illustrated in FIGS. 1 and 2, the master system 100 may include aninput unit 110 and a display unit 120.

The input unit 110 refers to an element that receives an instruction forselection of an operation mode of the surgical robot system, or aninstruction for remote control of operation of the slave system 200 bythe operator S. In the present embodiment, the input unit 110 mayinclude a haptic device, a clutch pedal, a switch, and a button, but isnot limited thereto. For example, a voice recognition device may beused. Hereinafter, a haptic device will be exemplarily described as anexample of the input unit 110.

FIG. 1 exemplarily illustrates that the input unit 110 includes twohandles 111 and 113, but the present embodiment is not limited thereto.For example, the input unit 110 may include one handle, or three or morehandles.

The operator S respectively manipulates the two handles 111 and 113using both hands, as illustrated in FIG. 1, to control an operation ofthe robot arm 210 of the slave system 200. Although not shown in detailin FIG. 1, each of the handles 111 and 113 may include a plurality oflinks and a plurality of joints.

A joint refers to a connection between two links and may have 1 degreeof freedom (DOF). Here, the term “degree of freedom (DOF)” refers to aDOF with regard to kinematics or inverse kinematics. A DOF of a deviceindicates the number of independent motions of a device, or the numberof variables that determine independent motions at relative positionsbetween links. For example, an object in a 3D space defined by X-, Y-,and Z-axes has at least one DOF selected from the group consisting of 3DOFs to determine a spatial position of the object with respect to aposition on each axis and 3 DOFs to determine a spatial orientation ofthe object with respect to a rotation angle relative to each axis. Morespecifically, it will be appreciated that when an object is movablealong each of X-, Y-, and Z-axes and is rotatable about each of X-, Y-,and Z-axes, it will be appreciated that the object has 6 DOFs.

In addition, a detector (not shown) may be mounted on the joint. Thedetector may detect data indicating the state of the joint, such asforce/torque data applied to the joint, position data of the joint, andspeed data. Accordingly, in accordance with manipulation of the inputunit 110 by the operator S, the detector (not shown) may detect data ofthe status of the manipulated input unit 110, and the controller 130generates a control signal corresponding to the data of the status ofthe input unit 110 detected by the detector (not shown) by use of acontrol signal generator 131 to transmit the generated control signal toa communication unit 250 in the slave system 200 via a communicationunit 140. That is, the controller 130 of the master system 100 maygenerate a control signal according to manipulation of the input unit110 by the operator S using the control signal generator 131 andtransmit the generated control signal to the slave system 200 via thecommunication unit 140.

The display unit 120 of the master system 100 may display an image of asurgical region (hereinafter, referred to as “affected area”) of thepatient P obtained by the image capture unit 310 of the imaging system300 which will be described later, a virtual image acquired byconverting a medical image of the patient P before surgery into a 3Dimage, and the like. To this end, the master system 100 may include animage processor 133 to receive image data from the imaging system 300and to output the image data to the display unit 120. In this regard,the “image data” may include an image of an affected image acquired bythe image capture unit 310, a virtual image acquired by converting amedical image of the patient P before surgery into a 3D image, and thelike as described above, but is not limited thereto.

The display unit 120 may include at least one monitor, and each monitormay be implemented to individually display data used for surgery. Forexample, when the display unit 120 includes three monitors, one of themonitors may display an affected area image acquired by the imagecapture unit 310, and the other two monitors may respectively display avirtual image acquired by converting a medical image of the patient Pbefore surgery into a 3D image, and data of operation of the slavesystem 200 and the patient P. In this regard, the number of monitors mayvary according to the type of data to be displayed.

Here, the term “data of the patient” may refer to information indicatingvital signs of the patient, such as bio-data including body temperature,pulse, respiration, and blood pressure, for example. In order to providesuch bio-data to the master system 100, the slave system 200, which willbe described later, may further include a bio-data measurement unitincluding a body temperature-measuring module, a pulse-measuring module,a respiration-measuring module, a blood pressure-measuring module, andthe like. To this end, the master system 100 may further include asignal processor (not shown) to receive bio-data from the slave system200, process the bio-data, and output the resultant data to the displayunit 120.

The slave system 200 may include a plurality of robot arms 210 andvarious surgical instruments 230 mounted on ends of the robot arms 210.The robot arms 210 may be coupled to a body 201 in a fixed state andsupported thereby, as illustrated in FIG. 1. In this regard, the numbersof the surgical instruments 230 and the robot arms 210 used at once mayvary according to various factors, such as diagnostic methods, surgicalmethods, and spatial limitations of an operating room.

In addition, each of the robot arms 210 may include a plurality of links211 and a plurality of joints 213. Each of the joints 213 may connectlinks 211 and may have 1 DOF or greater.

In addition, a drive unit 215 to control motion of the robot arm 210according to a control signal received from the master system 100 may bemounted on each of the joints of the robot arm 210. For example, whenthe operator S manipulates the input unit 110 of the master system 100,the master system 100 generates a control signal corresponding to thestatus data of the manipulated input unit 110, and transmits the controlsignal to the slave system 200, and a controller 240 of the slave system200 drives the drive unit 210 in accordance with the control signalreceived from the master system 100, to control motion of each joint ofthe robot arm 210. Here, a substantial control process, such as rotationand movement in a direction corresponding to the robot arm 210, inaccordance with manipulation of the input unit 110 by the operator Sdoes not fall within the scope of the present disclosure, and thus adetailed description thereof will not be given.

Meanwhile, each joint of the robot arm 210 of the slave system 200 maymove according to the control signal received from the master system 100as described above. However, the joint may also move by external force.That is, the assistant A positioned near the operating table maymanually move each of the joints of the robot arm 210 to control thelocation of the robot arm 210, or the like.

The surgical instruments 230, although not shown in detail in FIG. 1,may include a housing mounted on one end of the robot arm 210, a shaftextending from the housing by a predetermined length, and an endeffector mounted on one end of the shaft.

In generally, the surgical instruments 230 may be classified into mainsurgical instruments and auxiliary surgical instruments. Here, the term“main surgical instruments” may refer to surgical instruments includingend effectors, such as a scalpel or surgical needle which performsdirect surgical motion, such as cutting, suturing, cauterizing, andrinsing, on the surgical region. The term “auxiliary surgicalinstruments” may refer to surgical instruments including end effectors,such as a skin holder which do not perform direct motions on thesurgical region and assist motion of the main surgical instruments.

In addition, the end effector that constitutes the surgical instrument230 and is directly applied to the affected area of the patient P mayinclude a clamp, a skin holder, a suction, a scalpel, scissors, agrasper, a surgical needle, a needle holder, a stapler, a cutting blade,and the like, but is not limited thereto. Any known instruments used forsurgery may also be used.

In addition, a drive wheel may be coupled to the housing. The endeffector may be driven by connecting the drive wheel with the endeffector via wire or the like and rotating the drive wheel. To this end,a drive unit (not shown) to rotate the drive wheel may be mounted on oneend of the robot arm 210. For example, upon manipulation of the inputunit 110 of the master system 100 by the operator S, the master system100 generates a control signal corresponding to the status of themanipulated input unit 110 and transmits the control signal to the slavesystem 200, and the controller 240 of the slave system 200 drives thedrive unit (not shown) in accordance with the control signal receivedfrom the master system 100, to control the end effector in a desiredmanner. However, the operating mechanism of the end effector is notnecessarily constructed as described above, and various otherelectrical/mechanical mechanisms to realize motions of the end effectorfor robot surgery may be applied.

In addition, the slave system 200 according to the present embodimentmay further include a position sensor 217 to detect locations of thesurgical instruments 230 as illustrated in FIG. 2. The position sensor217 may be a potentiometer, an encoder, or the like, but is not limitedthereto.

The position sensor 217 may be mounted on each joint of the robot arm210 provided with the surgical instrument. The position sensor 217detects data regarding the status of motion of each joint of the robotarm 210. The controller 240 receives the detected data from the positionsensor 217. The position and direction of the surgical instruments 230may be calculated by use of a position calculator 241. In this regard,the position calculator 241 applies the input data to kinematics of therobot arm 210 to calculate the position and direction of the surgicalinstruments 230. In addition, the controller 240 may transmit thecalculated data regarding the position and direction of the surgicalinstruments 230 to the imaging system 300, which will be describedlater.

As described above, the position and direction of each of the surgicalinstruments 230 may be estimated by detecting the status data of each ofthe joints of the robot arms 210 provided with the surgical instruments230. As a result, the position and direction of the surgical instrument230 may be efficiently estimated even when the surgical instrument 230is located outside of a field of vision of the image capture unit 310,or a field of vision of the image capture unit 310 is blocked due tointernal organs or the like.

In addition, the slave system 200 may include a display unit 220 thatdisplays an affected area image acquired by the image capture unit 310of the imaging system 300, a virtual image acquired by converting amedical image of the patient P before surgery into a 3D image, and thelike in the same manner as in the master system 100. The slave system200 may include an image processor 243 to receive image data from theimaging system 300 and to output the image data to the display unit 220.

According to the present embodiment as illustrated in FIG. 2, theimaging system 300 may include an image capture unit 310 that acquires aplurality of images of an affected area, an image generator 330 thatdetects an occluded region in each of the affected area images acquiredby the image capture unit 310, removes the occluded region therefrom,warps each of the affected area images from which the occluded region isremoved, and matches the resultant images to generate a final image, anda controller 320 that operates the image capture unit 310 to acquire theplurality of affected area images and inputs each of the acquiredaffected area images to the image generator 330 to generate the finalimage.

In the illustrated embodiment, the term “occluded region” may refer to aregion of the affected area obscured by an object used on the patient P.Here, the “object” may be a surgical instrument 230, gauze, and thelike, but is not limited thereto. That is, the “occluded region” mayindicate a region of the affected area that is not readily visible.

In addition, referring to FIG. 2, the imaging system 300 according tothe illustrated embodiment includes the master system 100 and the slavesystem 200 which are separately arranged from each other. However, thisstructure is exemplarily illustrated for convenience of explanation. Theimaging system 300 may be integrated with the slave system 200.Alternatively, the image capture unit 310 of the imaging system 300 maybe disposed in the slave system 200, and the controller 320 and theimage generator 330 of the imaging system 300 may be disposed in themaster system 100. However, the present embodiment is not limitedthereto, and may be implemented in various ways.

In the illustrated embodiment, the image capture unit 310 of the imagingsystem 300 may acquire a plurality of affected area images. To this end,the image capture unit 310 of the surgical robot system may include aplurality of cameras including a first camera 312 and a second camera315 as illustrated in FIG. 3. In this regard, the image capture unit 310including two cameras is exemplarily illustrated in FIG. 3 fordescriptive convenience, but the number of cameras contained in theimage capture unit 310 is not limited thereto. In addition, the imagecapture unit 310 may further include a first drive unit 313 and a seconddrive unit 316 respectively driving the first camera 312 and the secondcamera 315, and the number of drive units may correspond to the numberof cameras.

In addition, the first camera 312 and the second camera 315 may bemounted on a support member 317 as illustrated in FIG. 5. In thisregard, a linear support member 317 is exemplarily illustrated in FIG.5. However, the shape of the support member 317 is not limited theretoand may be embodied in various shapes, as necessary.

In addition, according to the present embodiment as illustrated in FIG.5, the first camera 312 and the second camera 315 mounted on the supportmember 317 may respectively move along the support member 317 indirections designated by arrows. To this end, the image capture unit 310may further include a first moving unit 311 and the second moving unit314 respectively moving the first camera 312 and the second camera 315,and the number of moving units may correspond to the number of cameras.

Meanwhile, according to the illustrated embodiment, the first and secondcameras 312 and 315 of the image capture unit 310 are movably mounted onthe support member 317 as described above. However, the first and secondcameras 312 and 315 may also be fixed to the support member 317.

In addition, in the illustrated embodiment, the first camera 312 and thesecond camera 315 may be depth cameras, but are not limited thereto. Inthis regard, the term “depth camera” refers to a camera to calculate adistance to an object by radiating laser beams or infrared (IR) light,for example, to the object or a target region (herein, an occludedregion) and receiving the reflected laser beams or IR light, therebyestimating depth information of the object or the target region. A highresolution image may be acquired by use of such depth cameras, and depthof each pixel may be estimated. Thus, depth cameras may be applied to amoving object or 3D modeling.

Meanwhile, the first camera 312 and the second camera 315 may be generaluse cameras, such as complementary metal-oxide semiconductor (CMOS)cameras and charge coupled devices (CCDs), for example, in addition tothe depth cameras. In this case, the controller 320 of the imagingsystem 300 estimates depth information from each of the first and secondcameras 312 and 315 to the occluded regions. Here, depth information maybe estimated by use of sum of absolute differences (SAD) or sum ofsquared differences (SSD). However, the present embodiment is notlimited thereto, and any known method of estimating depth informationmay also be applied thereto. SAD and SSD are well known techniques inthe art, and thus a detailed description thereof will not be given.

The controller of the imaging system 300 according to the illustratedembodiment, estimates depth information from each of the first andsecond cameras 312 and 315 to the occluded regions as described above,to determine whether to move each of the first and second cameras 312and 315 from the current position to another position, i.e., a positioncorresponding to the estimated depth information. In accordance with theposition of the occluded regions whether the occluded regions arepositioned close to the first and second cameras 312 and 315 or close tothe affected area, each of the first and second cameras 312 and 315 maybe moved. Thus, a plurality of affected area images in which thepositions of the occluded regions do not overlap each other may beacquired.

For example, as illustrated in FIG. 6, when the surgical instrument 230causing the occluded region is located between each of the first andsecond cameras 312 and 315 and the affected area, an affected area imageacquired by the first camera 312 has a first occluded region caused bythe surgical instrument 230, and an affected area image acquired by thesecond camera 315 has a second occluded region. Here, the first andsecond occluded regions respectively contained in the affected areaimages acquired by the first and second cameras 312 and 315 aredifferent from each other in positions and do not overlap each other.

In addition, as illustrated in FIG. 7, when the surgical instrument 230is positioned close to the first and second cameras 312 and 315, anaffected area image acquired by the first camera 312 has a firstoccluded region, but an affected area image acquired by the secondcamera 315 does not have an occluded region.

As described above, when the surgical instrument 230 is positionedbetween each of the first and second cameras 312 and 315 and theaffected area or close to the first and second cameras 312 and 315,occluded regions in the affected area images acquired by the first andsecond cameras 312 and 315 do not overlap each other. Thus, there is noneed to move the first and second cameras 312 and 315.

On the other hand, as illustrated in FIG. 8, when the surgicalinstrument 230 is positioned close to the affected area, a firstoccluded region contained in an affected area image acquired by thefirst camera 312 overlaps a second occluded region contained in anaffected area image acquired by the second camera 315 by a portion a.Because the overlap portion (portion a) of the occluded region isneither visible in the affected area image acquired by the first camera312 nor in the affected area image acquired by the second camera 315, itis impossible to restore the overlap portion a. Accordingly, asillustrated in FIG. 9, the second camera 315 is moved in a directiondesignated by an arrow, so that the first occluded region does notoverlap the second occluded region.

That is, according to the present embodiment, a plurality of affectedarea images in which the positions of the occluded regions do notoverlap each other may be acquired by estimating depth information fromeach of the first and second cameras 312 and 315 to the surgicalinstrument 230 and moving the first and second cameras 312 and 315 tocorresponding positions in accordance with the estimated depthinformation.

Although the case in which the surgical instrument 230 is positionedbetween each of the first and second cameras 312 and 315 and theaffected area has been described, this is an example for descriptiveconvenience, and the occluded region may also be caused by various otherfactors in affected area images. It will be apparent to those havingordinary skill in the art that the present embodiment is applicable tovarious other cases.

In addition, the image generator 330 of the imaging system 300 accordingto the present embodiment may include an image processor 331, an imagewarping unit 333, and an image matching unit 335, as illustrated in FIG.4.

The image processor 331 may perform image processing to detect occludedregions contained in a plurality of affected area images acquired by theimage capture unit 310, and remove the detected occluded regions. Inthis regard, the occluded regions contained in each of the affected areaimages may be detected by use of various methods. For example, anoccluded region caused by the surgical instrument 230 may be detected byestimating position and direction of the surgical instrument 230.

In general, the position and direction of the surgical instrument 230that is inserted into and operates upon the patient P may be estimatedusing various methods as follows. First, the position and direction ofthe surgical instrument 230 may be calculated by detecting status dataof the joint of the robot arm 210 using the position sensor 217 attachedto the joint of the robot arm 210 provided with the surgical instrument230 and applying the detected data to inverse kinematics. Second, theposition and direction of the separated surgical instrument 230 may becalculated by attaching a predetermined marker to the surgicalinstrument 230 and separating the surgical instrument 230 from thebackground via recognition of the predetermined marker in the imageacquired by the image capture unit 310. Third, the position anddirection of the surgical instrument 230 may be calculated by extractinga blob closest to the camera, i.e., having the smallest depth data,using depth data under the condition that the bottom of a screenindicates a position closest to the camera, and spreading the extractedblob through a region growing method. In this regard, the presence ofthe surgical instrument 230 on the screen may be determined by use of apredetermined threshold. “Region growing” is a method of partitioningthe image into basic regions having common properties and continuouslyintegrating from regions having similar properties to regions having awider variety of properties.

Such methods of detecting the surgical instrument 230 are well known inthe art, and thus a detailed description thereof will not be given. Inaddition, two or more of the three methods may be simultaneously used toaccurately calculate the position or direction of the surgicalinstrument 230, but the present embodiment is not limited thereto.

As described above, the occluded region is detected in each of theaffected area images, and the detected occluded region is removed. Inthis regard, a portion of the affected area image from which theoccluded region is removed becomes a blank region. The removed portionsdo not overlap each other and are located at different positions indifferent affected area images. That is, the position of the removedportion in the image acquired by the first camera 312 is different fromthat of the removed portion in the image acquired by the second camera315. Accordingly, the removed portion of the image acquired by the firstcamera 312 may be replaced with a portion of the image acquired by thesecond camera 315 corresponding thereto. The removed portion of theimage acquired by the second camera 315 may be replaced with a portionof the image acquired by the first camera 312 corresponding thereto.

The image warping unit 333 may warp a plurality of images from which theoccluded regions are removed by the image processor 331. In this regard,the term “warping” refers to a method of partially or entirelytransforming the image. Generally, “warping” may include transformationmethods such as affine transformation, perspective transformation, andbilinear transformation, for example, but the method is not limitedthereto. These methods are well known in the art, and thus a detaileddescription thereof will not be given.

That is, according to the illustrated embodiment, the first camera 312and the second camera 315 are disposed to acquire a plurality ofaffected area images. The first camera 312 and the second camera 315capture affected area images while being spaced apart from each other bya predetermined distance. As described above, the affected area imagesacquired by the first camera 312 and the second camera 315 disposed atdifferent positions may have different forms. In this regard, the“different forms” may be obtained according to the position and angle ofthe camera. For example, when the first camera 312 and the second camera315 are linearly disposed and spaced apart from each other by apredetermined distance, an affected area image acquired by the firstcamera 312 may have a region, which is smaller and distorted compared toa region of an affected area image acquired by the second camera 315corresponding thereto. An affected area image acquired by the secondcamera 315 may have a region, which is smaller and distorted compared toa region of an affected area image acquired by the first camera 312corresponding thereto. That is, the affected area images respectivelyacquired by the first and second cameras 312 and 315 may have differentpositions and degrees of distortion. However, this is an exemplaryembodiment, and a plurality of affected area images having differentdegrees of distortion may be acquire according to the position and angleof the cameras.

In the illustrated embodiment, a final image is generated by matchingall of the affected area images acquired by the first and second cameras312 and 315. Thus, the affected area images having different forms aretransformed to have a common form. This process may be performed by theimage warping unit 333.

The image matching unit 335 matches all of the affected area imageswarped by the image warping unit 333 to produce a final image in whichthe occluded regions are restored. In this regard, image matching may beperformed by various methods known in the art. For example, imagematching may be performed by local feature matching by which images arematched after aligning coinciding points, or a Harris Corner detectionby which images are matched to 3D spatial coordinates using external andinternal parameters of a camera, and the like. However, the presentembodiment is not limited thereto.

As described above, the image generator 330 of the imaging system 300may generate a final image in which the occluded regions are restored bydetecting occluded regions in each of the affected area images acquiredby the first camera 312 and the second camera 315, removing the occludedregions therefrom, warping each of the affected area images from whichthe occluded regions are removed, and matching the warped images. Thatis, the removed portion, i.e., the occluded region, of the affected areaimage acquired by the first camera 312 is filled with a correspondingregion of the affected area image acquired by the second camera 315. Theremoved region, i.e., the occluded region, of the affected area imageacquired by the second camera 315 is filled with a corresponding regionof the affected area image acquired by the first camera 312.Accordingly, an affected area image in which the occluded region isrestored in real time may be generated.

In addition, in the illustrated embodiment, the imaging system 300 mayfurther include a communication unit 340. The controller 320 of theimaging system 300 may transmit the final image generated by theaforementioned image generator 330, i.e., the affected area image inwhich the occluded regions are restored, to the master system 100 andthe slave system 200 via the communication unit 340. The master system100 and the slave system 200 may display the received final image ondisplay units 120 and 220, respectively.

FIG. 10 is a flowchart sequentially illustrating a method of controllinga surgical robot system.

First, a plurality of affected area images is acquired using a pluralityof cameras (operation S1010). In this regard, the cameras may be mountedon the support member 317 to be spaced apart from each other by apredetermined distance, as illustrated in FIG. 5. In FIG. 5, twocameras, i.e., the first and second cameras 312 and 315, areillustrated, but the number of cameras is not limited thereto. Inaddition, a linear support member 317 is illustrated in FIG. 5. However,this is an exemplary embodiment, and the support member 317 may havevarious shapes, as necessary. In addition, the cameras may beimplemented to be movable along the support member 317 as illustrated inFIG. 5, but are not limited thereto. The cameras may be fixed to thesupport member 317.

A plurality of affected area images acquired by the plurality ofcameras, which are disposed spaced apart from each other by apredetermined distance, may have various forms. That is, because theinside of the patient P does not have a planar structure but a 3Dstructure, and the cameras are disposed at different positions, theaffected area images acquired by each of the cameras may have differentforms.

Then, an occluded region is detected from each of the acquired affectedarea images (operation S1020). In the illustrated embodiment, the term“occluded region” may refer to a region of the affected area obscured byan object inserted into, or used for, the patient P. Here, the “object”may be a surgical instrument 230, gauze, and the like, but is notlimited thereto. That is, “occluded region” may indicate a region of theaffected area that is not readily visible. In addition, the occludedregions may be formed at different positions of the affected area imagesand do not overlap each other. For example, as illustrated in FIG. 6,because the first camera 312 and the second camera 315 are disposed tobe spaced apart from each other by a predetermined distance, the firstoccluded region and the second occluded region may not overlap eachother in the affected area images acquired by the first and secondcameras 312 and 315.

However, the occluded regions of the affected area images may overlapeach other according to depth of an object causing the occluded regions.The overlapping of the occluded regions may be removed by moving thecamera, which will be described in detail later.

In addition, a variety of known methods may be used to detect theoccluded region according to the illustrated embodiment. For example,when the occluded region is caused by the surgical instrument 230, theoccluded region may be detected by estimating the current position anddirection of the surgical instrument 230. In general, the currentposition and direction of the surgical instrument 230 that is insertedinto the patient P and operates thereupon may be detected by thefollowing methods. First, the position and direction of the surgicalinstrument 230 may be calculated by detecting status data of the jointof the robot arm 210 using the position sensor 217 attached to the jointof the robot arm 210 provided with the surgical instrument 230 andapplying the detected data to inverse kinematics. Second, the positionand direction of the separated surgical instrument 230 may be calculatedby attaching a predetermined marker to the surgical instrument 230 andseparating the surgical instrument 230 from the background viarecognition of the predetermined marker in the image acquired by theimage capture unit 310. Third, the position and direction of thesurgical instrument 230 may be calculated by extracting a blob closestto the camera, i.e., having the smallest depth data, using depth dataunder the condition that the bottom of a screen indicates a positionclosest to the camera, and spreading the extracted blob through a regiongrowing method. In this regard, the presence of the surgical instrument230 on the screen may be determined by use of a predetermined threshold.“Region growing” is a method of partitioning the image into basicregions having common properties and continuously integrating fromregions having similar properties to regions having a wider variety ofproperties.

Such methods of detecting the surgical instrument 230 are well known inthe art, and thus a detailed description thereof will not be given. Inaddition, two or more of the three methods may be simultaneously used toaccurately calculate the position or direction of the surgicalinstrument 230, but the present embodiment is not limited thereto. Inaddition, a method of detecting the occluded region caused by thesurgical instrument 230 is described herein. However, this is anexemplary embodiment, and various known methods for detecting theoccluded region may also be used according to the cause of occlusion.

When the occluded regions are not detected in this operation, theaffected area images are respectively warped (operation S1040), and thewarped affected area images are matched to generate a final image(operation S1050).

Then, the occluded regions are respectively removed from the affectedarea images (operation S1030). Here, because the positions of theoccluded regions of the affected area images do not overlap each otheras described above, regions of the affected area images from which theoccluded regions are removed do not overlap each other, either. Theseregions will be filled with corresponding regions of another image in asubsequent operation.

Then, each of the affected area images from which the occluded region isremoved is warped (operation S1040). Here, warping is a method ofpartially or entirely transforming the image. Generally, warping mayinclude transformation methods such as affine transformation,perspective transformation, and bilinear transformation, for example,but the method is not limited thereto. These methods are well known inthe art, and thus a detailed description thereof will not be given.

That is, according to the illustrated embodiment, the first camera 312and the second camera 315 are disposed to acquire a plurality ofaffected area images. The first camera 312 and the second camera 315capture affected area images in a state of being spaced apart from eachother by a predetermined distance. As described above, the affected areaimages acquired by the first camera 312 and the second camera 315disposed at different positions may have different forms. In theillustrated embodiment, a final image is generated by matching all ofthe affected area images acquired by the first and second cameras 312and 315. Thus, the affected area images having different forms need tobe transformed to have a common form.

Then, the warped affected area images are matched to produce a finalimage (operation S1050). In this regard, image matching may be performedby various methods known in the art. For example, image matching may beperformed by local feature matching by which images are matched afteraligning coinciding points, or a Harris Corner detection by which imagesare matched to 3D spatial coordinates using external and internalparameters of a camera, and the like. However, the present embodiment isnot limited thereto.

As described above, a final image in which the occluded regions arerestored may be generated according to the method of controlling thesurgical robot system according to the illustrated embodiment bydetecting occluded regions in each of the affected area images acquiredby the first camera 312 and the second camera 315, removing the occludedregions therefrom, warping each of the affected area images from whichthe occluded regions are removed, and matching the warped images. Thatis, the removed portion, i.e., the occluded region, of the affected areaimage acquired by the first camera 312 is filled with a correspondingregion of the affected area image acquired by the second camera 315. Theremoved region, i.e., the occluded region, of the affected area imageacquired by the second camera 315 is filled with a corresponding regionof the affected area image acquired by the first camera 312.Accordingly, an affected area image in which the occluded region isrestored in real time may be generated.

Meanwhile, as described above, the occluded regions of the affected areaimages acquired by the first camera 312 and the second camera 315 mayoverlap each other according to depth of an object causing occlusion.

For example, as illustrated in FIG. 8, when the surgical instrument 230inserted into the patient P is positioned close to the affected area, afirst occluded region contained in an affected area image acquired bythe first camera 312 overlaps a second occluded region contained in anaffected area image acquired by the second camera 315 by a portion a.Because the overlap portion (portion a) of the occluded region isneither visible in the affected area image acquired by the first camera312 nor in the affected area image acquired by the second camera 315, itis impossible to restore the overlap portion a. Accordingly, asillustrated in FIG. 9, the second camera 315 is moved in a directiondesignated by an arrow, so that the first occluded region does notoverlap the second occluded region.

To this end, according to the illustrated embodiment as illustrated inFIG. 11, the occluded regions are respectively detected from theaffected area images (operation S1020), and depth information from thefirst and second cameras 312 and 315 to the detected occluded regions isestimated (operation S1021). Here, depth information may be estimated byuse of sum of absolute differences (SAD) or sum of squared differences(SSD). However, the present embodiment is not limited thereto, and anyknown method of estimating depth information may also be appliedthereto. SAD and SSD are well known techniques in the art, and thus adetailed description thereof will not be given.

In addition, as another method of estimating depth information, a methodof estimating depth information using a depth sensor disposed at each ofthe first and second cameras 312 and 315 may be employed. In thisregard, the term “depth sensor” refers to a sensor to calculate adistance to an object by radiating laser beams or infrared (IR) light,for example, to the object (herein, occluded region), depth of whichwill be estimated, and receiving the reflected laser beams or IR light,thereby estimating depth information of the object.

Then, it is determined whether the estimated depth information is withina predetermined range (operation S1023). When the estimated depthinformation is within the predetermined range, the occluded regionsdetected from the affected area images are removed (operation S1030).When the estimated depth information is not within the predeterminedrange, the first camera 312 and the second camera 315 are moved topositions corresponding to the estimated depth information, and thenaffected area images are acquired at the moved positions (operationS1025). Then, occluded regions are detected from the re-acquiredaffected area images (operation S1027). Then, the detected occludedregions are respectively removed from the affected area images(operation S1030).

As described above, depth information from each of the first and secondcameras 312 and 315 to the occluded region is estimated, and theaffected area images are acquired by moving the first and second cameras312 and 315 according to the estimated depth information. As a result, aplurality of affected area images in which the positions of the occludedregions do not overlap each other may be acquired.

The above-described embodiments may be recorded in computer-readablemedia including program instructions to implement various operationsembodied by a computer. The media may also include, alone or incombination with the program instructions, data files, data structures,and the like. The program instructions recorded on the media may bethose specially designed and constructed for the purposes ofembodiments, or they may be of the kind well-known and available tothose having skill in the computer software arts. Examples ofcomputer-readable media include magnetic media such as hard disks,floppy disks, and magnetic tape; optical media such as CD ROM disks andDVDs; magneto-optical media such as optical disks; and hardware devicesthat are specially configured to store and perform program instructions,such as read-only memory (ROM), random access memory (RAM), flashmemory, and the like. The computer-readable media may also be adistributed network, so that the program instructions are stored andexecuted in a distributed fashion. The program instructions may beexecuted by one or more processors. The computer-readable media may alsobe embodied in at least one application specific integrated circuit(ASIC) or Field Programmable Gate Array (FPGA), which executes(processes like a processor) program instructions. Examples of programinstructions include both machine code, such as produced by a compiler,and files containing higher level code that may be executed by thecomputer using an interpreter. The above-described devices may beconfigured to act as one or more software modules in order to performthe operations of the above-described embodiments, or vice versa.

Although a few embodiments of the present invention have been shown anddescribed, it would be appreciated by those skilled in the art thatchanges may be made in these embodiments without departing from theprinciples and spirit of the invention, the scope of which is defined inthe claims and their equivalents.

What is claimed is:
 1. A method of controlling a robot, the methodcomprising: acquiring a plurality of affected area images using aplurality of cameras; detecting an occluded region present in each ofthe plurality of affected area images by detecting state data of eachjoint of a robot arm, the robot arm configured to mount a surgicalinstrument thereon; removing detected occluded region from each of theplurality of affected area images; warping each of the plurality ofaffected area images from which the occluded region is removed; andgenerating a final image by matching warped affected area images.
 2. Themethod according to claim 1, wherein detected occluded regions aredisposed at different positions in the plurality of affected areaimages.
 3. The method according to claim 1, wherein the occluded regionis a region at which the surgical instrument is located in each of theplurality of affected area images.
 4. The method according to claim 3,wherein the detecting of the occluded region is performed by attaching amarker to the surgical instrument, and detecting the marker from each ofthe plurality of affected area images.
 5. The method according to claim3, wherein the detecting of the occluded region is performed byextracting a blob closest to the plurality of cameras from the each ofthe plurality of affected area images and spreading an extracted blobusing a region growing method.
 6. The method according to claim 1,wherein the detecting of the occluded region is performed by estimatingdepth information from the plurality of cameras to an object obscuringeach of the plurality of affected areas.
 7. The method according toclaim 6, wherein the estimating of the depth information to the occludedregion is performed by use of sum of absolute differences (SAD) or sumof squared differences (SSD).
 8. The method according to claim 6,wherein the estimating of the depth information to the occluded regionis performed by use of a depth sensor attached to the plurality ofcameras.